Endoscope processor, training device, information processing method, training method and program

ABSTRACT

A processor for an endoscope or the like that assists an endoscopic examination using an appropriate reference image is provided. A processor for an endoscope includes an image acquisition unit that acquires an endoscope image; a region acquisition unit that inputs the endoscope image acquired by the image acquisition unit to a first learning model that outputs a target region corresponding to a predetermined region to be photographed when the endoscope image is input to acquire the target region; and an image output unit that outputs the endoscope image acquired by the image acquisition unit and an index indicating the target region acquired by the region acquisition unit with the endoscope image and the index superimposed.

TECHNICAL FIELD

The present invention relates to a processor for an endoscope, atraining device, an information processing method, and a program.

BACKGROUND ART

In an endoscopic examination in a complete medical checkup, a periodicmedical examination, or the like, it is necessary not only to discover alesion but also to observe the entire interior wall of an organ todetermine that no lesion is present. In the guidelines set by themedical society, each medical institution, and the like, a region or thelike whose image is recorded is set. By recording the image according tothe guidelines, it is possible to implement prevention of overlooking ofthe lesion and double check in which the endoscope specialist doctorconfirms the image at a later date.

An endoscope system that assists recording of an image according to theguidelines by determining similarity between an endoscope image beingcaptured and a reference image according to the guidelines has beenproposed (Patent Literature 1). In the endoscope system of PatentLiterature 1, an examination region being observed is determined basedon an insertion length of an insertion portion inserted into a medicalexaminee, thereby selecting a reference image corresponding to theexamination region.

CITATION LIST Patent Literature

-   Patent Literature 1: JP 2018-139848 A

SUMMARY OF INVENTION Technical Problem

However, for example, in the endoscopic examination of the stomach,since the endoscope is variously operated inside the stomach inflated bythe air, the insertion length does not correspond to the examinationregion. Also, in the endoscopic examination of the large intestine,since the intestinal tract is folded in a bellows shape outside theinsertion portion, the insertion length does not correspond to theexamination region. Therefore, in the endoscope system of PatentLiterature 1, it is difficult to select an appropriate reference imageand assist the doctor.

In an aspect, an object is to provide a processor for an endoscope orthe like that assists an endoscopic examination using an appropriatereference image.

Solution to Problem

A processor for an endoscope includes an image acquisition unit thatacquires an endoscope image, a region acquisition unit that inputs theendoscope image acquired by the image acquisition unit to a firstlearning model that outputs a target region corresponding to apredetermined region to be photographed when the endoscope image isinput to acquire the target region, and an image output unit thatoutputs the endoscope image acquired by the image acquisition unit andan index indicating the target region acquired by the region acquisitionunit with the endoscope image and the index superimposed.

Advantageous Effects of Invention

In an aspect, it is possible to provide a processor for an endoscope orthe like that assists an endoscopic examination using an appropriatereference image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is an explanatory diagram for explaining an outline of anautomatic imaging function.

FIG. 1B is an explanatory diagram for explaining an outline of anautomatic imaging function.

FIG. 1C is an explanatory diagram for explaining an outline of anautomatic imaging function.

FIG. 1D is an explanatory diagram for explaining an outline of anautomatic imaging function.

FIG. 2 is an explanatory diagram for explaining the configuration of theendoscope system.

FIG. 3 is an explanatory diagram for explaining a record layout of areference image DB.

FIG. 4 is an explanatory diagram for explaining a record layout of anexamination DB.

FIG. 5 is an explanatory diagram for explaining a configuration of afirst learning model.

FIG. 6 is an explanatory diagram for explaining a configuration of asecond learning model.

FIG. 7 is an explanatory diagram for explaining a screen displayexample.

FIG. 8 is an explanatory diagram for explaining a screen displayexample.

FIG. 9 is an explanatory diagram for explaining a screen displayexample.

FIG. 10 is an explanatory diagram for explaining a screen displayexample.

FIG. 11 is an explanatory diagram for explaining a screen displayexample.

FIG. 12 is an explanatory diagram for explaining a screen displayexample.

FIG. 13 is an explanatory diagram for explaining a screen displayexample.

FIG. 14 is an explanatory diagram for explaining a screen displayexample.

FIG. 15 is an explanatory diagram for explaining a screen displayexample.

FIG. 16 is an explanatory diagram for explaining a screen displayexample.

FIG. 17 is a flowchart for explaining a processing flow of a program.

FIG. 18 is a flowchart for explaining a processing flow of a subroutineof a target selection.

FIG. 19 is a flowchart for explaining a processing flow of a subroutineof a guide display.

FIG. 20 is a flowchart for explaining a processing flow of a subroutineof a record.

FIG. 21 is an explanatory diagram for explaining a screen displayexample of a second embodiment.

FIG. 22 is an explanatory diagram for explaining a screen displayexample of the second embodiment.

FIG. 23 is a flowchart for explaining a processing flow of thesubroutine of the target selection of the second embodiment.

FIG. 24 is a flowchart for explaining a processing flow of a subroutineof a guide display of the second embodiment.

FIG. 25 is an explanatory diagram for explaining a screen displayexample of a third embodiment.

FIG. 26 is an explanatory diagram for explaining a screen displayexample of the third embodiment.

FIG. 27 is a flowchart for explaining a processing flow of a subroutineof the target selection of the third embodiment.

FIG. 28 is an explanatory diagram for explaining a screen displayexample of a fourth embodiment.

FIG. 29 is an explanatory diagram for explaining the configuration of anendoscope system of a fifth embodiment.

FIG. 30 is an explanatory diagram for explaining a screen displayexample of the fifth embodiment.

FIG. 31 is a flowchart for explaining a processing flow of a program ofthe fifth embodiment.

FIG. 32 is an explanatory diagram for explaining a screen displayexample of a sixth embodiment.

FIG. 33 is an explanatory diagram for explaining a record layout of afollow-up observation DB.

FIG. 34 is a flowchart for explaining a processing flow of a program ofthe seventh embodiment.

FIG. 35 is a functional block diagram of a processor for an endoscope ofan eighth embodiment.

FIG. 36 is an explanatory diagram for explaining the configuration of anendoscope system of a ninth embodiment.

DESCRIPTION OF EMBODIMENTS First Embodiment

FIG. 1 is an explanatory diagram for explaining an outline of anautomatic imaging function. In the present embodiment, a case where anendoscopic examination of the stomach is performed in accordance withthe guidelines set by Japanese Society of Gastrointestinal CancerScreening will be described as an example. The guidelines may be setindependently in each medical institution or the like. The organ to beexamined may be a large intestine, duodenum, or the like.

The doctor inserts an endoscope 30 (see FIG. 2) into the medicalexaminee who receives the medical examination, and observes an endoscopeimage 49 in real time. FIG. 1A illustrates a state in which a region tobe photographed corresponding to a reference image defined in theguidelines is not included in the endoscope image 49. The doctorappropriately operates the endoscope 30 to direct the visual field tothe region to be photographed.

FIG. 1B illustrates a state in which a region to be photographed isincluded in the endoscope image 49. A portion corresponding to thereference image is surrounded by the guide frame (bounding box) 75. Asillustrated in FIGS. 1C and 1D, the doctor operates the endoscope 30 sothat the portion surrounded by the guide frame 75 spreads to the fullextent of the endoscope image 49.

As illustrated in FIG. 1D, when the guide frame 75 substantially matchesthe edge of the endoscope image 49, the endoscope image 49 is recordedas a still image. The still image may be generated by cutting out dataof one frame from the moving image being observed in real time, or maybe photographed using illumination and a shutter speed suitable for thestill image separately from the moving image.

The recording of the still image may be automatically performed or maybe performed based on an instruction from a doctor who has confirmed thestate of the guide frame 75. Thereafter, the doctor appropriatelyoperates the endoscope 30 toward the next region to be photographed.

By repeating the above operation, a series of still images correspondingto the reference image defined in the guidelines is recorded. In theprocess of operating the endoscope 30 so as to match the guide frame 75and the endoscope image 49, the doctor can sufficiently visually observethe region to be photographed. Therefore, it is possible to preventoverlooking of the lesion.

Since a series of endoscope images 49 is recorded at a position, anorientation, and a size in accordance with the reference image,so-called double check in which an endoscope specialist doctor or thelike confirms the recorded image can be efficiently performed.

Note that, in a case where a lesion is found during the endoscopicexamination, the doctor performs observation necessary for detaileddiagnosis out of the guidelines, and also performs endoscopic treatmentin some cases. In such a case, the guide frame 75 is unnecessary. Thedoctor records the appropriate endoscope image 49 based on professionaldetermination.

FIG. 2 is an explanatory diagram for explaining the configuration of anendoscope system 10. The endoscope system 10 includes the endoscope 30,the processor for an endoscope 20. The processor for an endoscope 20includes a control unit 21, a main storage device 22, an auxiliarystorage device 23, a communication unit 24, a speaker 25, a displaydevice interface (I/F) 26, an input device I/F 27, a reading unit 28, anendoscope connection unit 29, and a bus.

In FIG. 2, the illustration and description of the configuration thatrealizes the basic functions of the processor for an endoscope 20 suchas the light source, the air/water supply pump, the suction pump, andthe control unit of the image sensor provided at the distal end of theendoscope 30, are omitted.

The control unit 21 is an arithmetic control device that executes aprogram of the present embodiment. One or more central processing units(CPUs), graphics processing units (GPUs), or multi-core CPUs, and thelike are used for the control unit 21. The control unit 21 is connectedto each hardware unit constituting the processor for an endoscope 20 viaa bus.

The main storage device 22 is a memory device such as a static randomaccess memory (SRAM), a dynamic random access memory (DRAM), and a flashmemory. The main storage device 22 temporarily stores informationnecessary in the middle of processing performed by the control unit 21and a program being executed by the control unit 21.

The auxiliary storage device 23 is a memory device such as an SRAM, aflash memory, or a hard disk. The auxiliary storage device 23 includes afirst learning model 61, a second learning model 62, a reference imageDB 66, an examination DB 67, a program to be executed by the controlunit 21, and various data necessary for executing the program.

The first learning model 61, the second learning model 62, the referenceimage DB 66, and the examination DB 67 may be stored in an external massstorage device connected to the processor for an endoscope 20.

The communication unit 24 is an interface that performs datacommunication between the processor for an endoscope 20 and a hospitalinformation system (HIS) or the like. The speaker 25 outputs sound basedon an instruction from the control unit 21.

The display device I/F 26 is an interface that connects the processorfor an endoscope 20 and a display device 16. The input device I/F 27 isan interface that connects the processor for an endoscope 20 and aninput device such as a keyboard 17. Note that the display device 16 andthe input device may be configured integrally with the processor for anendoscope 20. The plurality of display devices 16 and the input devicemay be connected to the processor for an endoscope 20.

The endoscope 30 includes an insertion portion 31, an operation unit 32,a universal cord 33, and a connector unit 34. The operation unit 32includes a bending knob 35, three switch buttons 36, a channel inlet 37,an air/water supply button 321, and a suction button 322. A forceps plug371 having an insertion port for inserting a treatment instrument or thelike is fixed to the channel inlet 37.

The insertion portion 31 is long and has one end connected to theoperation unit 32 via a bend preventing portion. The insertion portion31 includes a soft portion 311, a bending section 312, and a distal endportion 313 in this order from the operation unit 32. The distal endportion 313 is provided with an observation window and an illuminationwindow, and an image sensor is disposed behind the observation window.The bending section 312 is bent according to an operation of the bendingknob 35.

Hereinafter, the longitudinal direction of the insertion portion 31 isreferred to as an “insertion direction”. Similarly, a side close to theoperation unit 32 along the insertion direction is referred to as anoperation unit side, and a side distant from the operation unit 32 isreferred to as a distal end side.

The universal cord 33 is long, and has a first end connected to theoperation unit 32 and a second end connected to the connector unit 34.The universal cord 33 is soft. The connector unit 34 is detachablycoupled to the endoscope connection unit 29 provided in the processorfor an endoscope 20.

The switch button 36 is connected to the processor for an endoscope 20via a cable (not illustrated) and the connector unit 34. Among variousfunctions of the processor for an endoscope 20, necessary operationsduring the endoscopic examination can be assigned to the switch buttons36. By using the switch buttons 36, the doctor can perform a necessaryoperation without releasing the hand from the endoscope 30.

In the present embodiment, the switch button 36 to which a so-calledrelease function of recording the endoscope image 49 as a still image isallocated is described as an image capturing button. When the doctor asthe user presses the image capturing button, the control unit 21promptly records the still image in the examination DB 67. In thefollowing description, recording a still image in examination DB 67 maybe referred to as “photographing”.

FIG. 3 is an explanatory diagram for explaining a record layout of thereference image DB 66. The reference image DB 66 is a DB in which thephotographing sequence according to the guidelines, the region name, andthe reference image are recorded in association with each other. Thereference image DB 66 includes a photographing sequence field, a regionname field, and a reference image field.

In the photographing sequence field, the photographing sequenceaccording to the guidelines is recorded. In the region name field, thename of the region to be photographed determined for capturing an imageaccording to the guidelines is recorded. In the reference image field, areference image recommended in the guidelines is recorded. The referenceimage DB 66 has one record for one region to be photographed.

FIG. 4 is an explanatory diagram for explaining a record layout of theexamination DB 67. The examination DB 67 is a DB in which the name ofthe region to be photographed obtained by capturing an image in theendoscopic examination, similarity between the captured image and thereference image, and the recorded image are recorded in association witheach other. The examination DB 67 includes a doctor identifier (ID)field, a medical examinee ID field, a date field, a number field, aregion name field, a similarity field, and an image field.

In the doctor ID field, a doctor ID uniquely assigned to a doctor incharge of the endoscopic examination is recorded. In the medicalexaminee ID field, a medical examinee ID uniquely assigned to a medicalexaminee who has undergone an endoscopic examination is recorded. A dateis recorded in the date field. In the number field, numbers assigned inascending order to the still images recorded during the endoscopicexamination are recorded.

In the region name field, the name of the region to be photographeddefined in the guidelines is recorded. “-” indicates the image capturedby the determination of the doctor separately from the guidelines andthe reference image. The similarity with the reference image is recordedin the similarity field. “-” means that the similarity is not calculatedbecause there is no reference image. In the image field, a file of stillimages is recorded.

The outline of the similarity recorded in the similarity field will bedescribed. The similarity is a numerical value obtained by quantifyingthe degree of similarity between two images. In the present embodiment,the similarity is expressed by a numerical value from 0 to 100, wherethe degree is 0 when there is no match and 100 when there is a completematch, but the similarity is not limited thereto.

Various methods have been conventionally used to quantify the similaritybetween two images. For example, for each of two images, a featureamount vector is generated based on various feature amounts such ascolor distribution and positions of feature points. The larger thedistance between the feature amount vectors, the lower the similarity,and the smaller the distance between the feature vectors, the higher thesimilarity.

The feature amount vector may be generated using a learning model forimage classification. As the learning model for image classification,for example, a learned convolutional neural network (CNN) model such asthe VGG 16 or the VGG 19 can be used. A learned model obtained by beingcaused to additionally learn the endoscope image 49 for these CNN modelsmay be used. The two images are input to the learning model for imageclassification, and an output from each node of the output layer is usedas a feature amount vector of each image.

When the endoscope image 49 is input, the similarity may be calculatedusing a learning model in which a parameter is adjusted so as to outputthe similarity with the reference image. For example, a probability thatthe reference image and the endoscope image 49 are the identical imageis used as the reference degree. In this case, a learning model forsimilarity calculation is generated for each reference image.

The similarity may be calculated based on a bounding box indicating apredetermined target extracted from the endoscope image 49 using anobject detection method such as regions with convolutional neuralnetworks (R-CNN). For example, it is determined that the similarity ishigh when the area in which the bounding box which extracts the cardiacregion overlaps with the bounding box indicating the cardiac region onthe reference image is large.

A probability when a predetermined target is detected using an objectdetection method such as the R-CNN may be used as the similarity.

The data recorded in the examination DB 67 may be appropriately uploadedto an electronic medical record system or the like.

FIG. 5 is an explanatory diagram for explaining a configuration of thefirst learning model 61. The first learning model 61 is a learning modelthat receives an input of the endoscope image 49 to output, a regionsimilar to a region to be photographed defined in the guidelines, thatis the reference image. FIG. 5 illustrates an example of the firstlearning model 61 using the R-CNN.

The first learning model 61 includes a region candidate extraction unit651, a classification unit 652, and a neural network (not illustrated).The neural network includes a convolutional layer, a pooling layer, anda fully-connected layer. The endoscope image 49 is input to the firstlearning model 61.

The region candidate extraction unit 651 extracts, from the inputendoscope image 49, region candidates of various sizes are extracted.The classification unit 652 calculates a feature amount of the extractedregion candidate, and classifies the object appearing in the regioncandidate based on calculated feature amount. The target to beclassified here is each region to be photographed defined in theguidelines. The first learning model 61 repeats extraction andclassification of region candidates, and determines a target regionincluding a region to be photographed appearing in each portion of theendoscope image 49 for which an input has been received.

In FIG. 5, the first learning model 61 outputs information about atarget region indicated by three detection frames (bounding boxes) 653.In the example illustrated in FIG. 5, a target region in which thelesser curvature of antrum appears with a probability of 80%, a targetregion in which the greater curvature of antrum appears with aprobability of 70%, and a target region in which the anterior wall ofantrum appears with a probability of 60% are output.

An outline of a method of generating the first learning model 61 will bedescribed. The first learning model 61 is generated using a plurality ofpieces of training data in which the endoscope image 49 in which alesion does not appear and the name, position, and range of the regionto be photographed included in the endoscope image 49 are recorded inassociation with each other. Specifically, when the endoscope image 49in the training data is input, the parameters of the neural network areadjusted by an error back propagation method so that the data outputfrom the classification unit 652 matches the name, position, and rangeof the region to be photographed in the training data.

FIG. 6 is an explanatory diagram for explaining a configuration of thesecond learning model 62. The second learning model 62 is a learningmodel that receives an input of the endoscope image 49 and outputs aregion in which the foreign matter such as mucus and residues appears.FIG. 6 illustrates an example of the second learning model 62 using theR-CNN.

The second learning model 62 includes the region candidate extractionunit 651, the classification unit 652, and a neural network (notillustrated). The neural network includes a convolutional layer, apooling layer, and a fully-connected layer. The endoscope image 49 isinput to the second learning model 62.

The region candidate extraction unit 651 extracts, from the inputendoscope image 49, region candidates of various sizes are extracted.The classification unit 652 calculates a feature amount of the extractedregion candidate, and classifies the object appearing in the regioncandidate based on calculated feature amount. The target to beclassified here is foreign matter such as mucus and residues. The secondlearning model 62 repeats extraction and classification of regioncandidates, and determines the foreign matter appearing in each portionof the endoscope image 49 that has received an input.

In FIG. 6, the second learning model 62 outputs information about theregions indicated by the two detection frames 653. In the exampleillustrated in FIG. 6, a region in which mucus appears with aprobability of 80%, and a region in which mucus appears with aprobability of 60% are output.

The second learning model 62 is created for each reference image. Beforephotographing the endoscope image 49 corresponding to the referenceimage, the control unit 21 uses the second learning model 62 todetermine whether there is a region in which the foreign matter appears.In a case where the foreign matter appears, the control unit 21 outputsa notification for calling user's attention. After a doctor who is auser removes the foreign matter using, for example, a sub water supplyfunction of the endoscope 30, the control unit 21 photographs a stillimage.

An outline of a method of generating the second learning model 62 willbe described. The second learning model 62 is generated by using aplurality of pieces of training data in which the endoscope image 49obtained by photographing the region to be photographed and the type,position, and range of the foreign matter appearing in the endoscopeimage 49 are recorded in association with each other. Specifically, whenthe endoscope image 49 in the training data is input, the parameters ofthe neural network are adjusted by an error back propagation method sothat the data output from the classification unit 652 matches the type,position, and range of the foreign matter in the training data.

Note that the second learning model 62 may be a learned model so as tooutput a region in which a lesion such as a cancer, an ulcer, a polyp,or a bleeding portion appears. In this case, the second learning model62 is generated using the training data in which the type, position, andrange of the lesion are recorded in association with each other inaddition to the foreign matter.

Note that, instead of the R-CNN, any object detection algorithm such asFast R-CNN, Faster R-CNN, Mask R-CNN, Single Shot Multibook Detector(SSD), or You Only Look Once (YOLO) may be used.

One learning model in which the first learning model 61 and the secondlearning model 62 are integrated may be used. In this case, the learningmodel receives an input of the endoscope image 49 to output both aregion similar to the reference image corresponding to the referenceimage and the foreign matter in the endoscope image.

FIGS. 7 to 16 are explanatory diagrams for explaining screen displayexamples. FIG. 7 illustrates an example of a screen displayed on thedisplay device 16 by the control unit 21 when a region including thenext region to be photographed is output by the first learning model 61.

The screen illustrated in FIG. 7 includes an endoscope image field 73, aprogress display field 71, a state display field 72, and a past imagefield 74. The control unit 21 displays the endoscope image 49 in theendoscope image field 73 in real time. The control unit 21 displays theguide frame 75 which is superimposed on the endoscope image 49. Theguide frame 75 is an index indicating a rough indication of a contour ofan image when the next region to be photographed is photographed withthe composition same as that of the reference image.

The control unit 21 displays the endoscope image 49 captured in the pastin the progress display field 71 as a thumbnail. In FIG. 7, thumbnailsare displayed in portions surrounded by thick frames. The portionsurrounded by the narrow frame is blank. That is, by displaying thethumbnail of the progress display field 71, the control unit 21functions as a first notification output unit that outputs anotification when the endoscope image 49 is recorded.

The reference image may be displayed in a portion surrounded by a narrowframe. In this case, the control unit 21 displays the captured endoscopeimage 49 and the reference image so as to be distinguishable from eachother by, for example, the thickness, color, or the like of a frame linesurrounding the periphery of the thumbnail.

When accepting the selection of the thumbnail, the control unit 21 popsup a large image. Since a method of generating thumbnails and displayingthe thumbnails in a list and a method of displaying the thumbnails in apop-up are conventionally used, the description thereof will be omitted.

The control unit 21 displays the operating mode in the state displayfield 72. “Guide Mode ON” means a mode for displaying the guide frame 75indicating the next region to be photographed. Even when the next regionto be photographed does not appear in the endoscope image 49 and theguide frame 75 is not displayed, the user can confirm the operating modeby viewing the state display field 72.

In the past image field 74, an endoscope image 49 captured immediatelybefore is displayed. Note that an image selected by the user from thethumbnails displayed in the progress display field 71 may be displayedin the past image field 74.

The user operates the endoscope 30 so that the guide frame 75 and theouter periphery of the endoscope image field 73 match with each other.With this operation, the user adjusts the endoscope image 49 tomagnification and layout similar to those of the reference image.

Note that the name and the like of the corresponding region to bephotographed may be displayed in the vicinity of the guide frame 75. Byreferring to arrows or the like from the four corners of the endoscopeimage 49 toward the corners of the guide frame 75, the displayindicating that the endoscope 30 should be operated to make the guideframe 75 large may be performed.

FIG. 8 is an example of a screen displayed instead of FIG. 7. On thescreen illustrated in FIG. 8, the control unit 21 displays a list of thenames of the regions to be photographed defined in the guidelines in theprogress display field 71. The control unit 21 displays a check box 713on the right side of each name. A checked check box 713 means that theimage has been captured. That is, by the display of the check box 713,the control unit 21 functions as a first notification output unit thatoutputs a notification when the endoscope image 49 is recorded.

The control unit 21 displays the name of the next region to bephotographed surrounded by a frame index 714. In the mode in which theguide frame 75 is not displayed, the control unit 21 does not displaythe frame index 714. Therefore, even when the state display field 72 isnot displayed, the user can confirm the operating mode.

The user can set whether to use the screen described with reference toFIG. 7 or the screen described with reference to FIG. 8. The controlunit 21 displays a screen based on selection by the user. Note that thescreen described with reference to FIGS. 7 and 8 is an example. Thecontrol unit 21 may display a screen of a design other than these.

FIG. 9 is an example of a screen in a state where the distal end portion313 is too close to the region to be photographed and the guide frame 75is not displayed. The control unit 21 indicates that the guide frame 75is outside the endoscope image field 73 by the arrow-like approach index751 displayed at the four corners of the endoscope image field 73.

Note that, instead of displaying the arrow, the control unit 21 maydisplay characters such as “too close” above the state display field 72,for example. It is possible to prevent the endoscope image 49 from beinghidden by the arrow.

FIG. 10 is an example of a screen in a case where the distal end portion313 approaches the region to be photographed but the endoscope image 49rotates with respect to the reference image. The control unit 21performs matching so that the similarity between the endoscope image 49inside the guide frame 75 and the reference image is high in a casewhere the guide frame 75 approaches the edge of the endoscope imagefield 73. When the endoscope image 49 is rotated with respect to thereference image, the guide frame 75 is displayed in a state of beingrotated with respect to the edge of the endoscope image field 73 asillustrated in FIG. 10.

The user performs a twisting operation of the insertion portion 31 whilebringing the distal end portion 313 close to the observation targetregion, thereby matching the guide frame 75 with the outer periphery ofthe endoscope image field 73.

FIG. 11 illustrates a modification of the screen in a case where theendoscope image 49 rotates with respect to the reference image similarlyto FIG. 10. Instead of displaying the guide frame 75 in a state of beingrotated with respect to the edge of the endoscope image field 73, thecontrol unit 21 displays the rotation index 752 indicating the rotationdirection with the index superimposed the endoscope image 49. The userperforms a twisting operation of the insertion portion 31 with referenceto the rotation index 752.

FIG. 12 is an example of a screen in a case where an accretion such asmucus exists in the region to be photographed. In a case where the guideframe 75 and the outer periphery of the endoscope image field 73substantially match with each other, the control unit 21 inputs theendoscope image 49 to the second learning model 62 and acquires a regionwhere an accretion exists. In a case where an accretion exists, thecontrol unit 21 displays an accretion frame 754 indicating the accretionwith the frame superimposed on the endoscope image 49. The control unit21 displays an accretion mark 762 instead of the past image field 74.The user removes the accretion using the sub water supply function orthe like of the endoscope 30.

FIG. 13 is an example of a screen being captured. The user matches theguide frame 75 with the outer periphery of the endoscope image field 73,confirms that there is no accretion, and then stops the distal endportion 313 to prevent image blurring. The control unit 21 displays aphotograph-in-progress index 761 and photographs the endoscope image 49.As described above, the endoscope image 49 without blurring isphotographed with the magnification and layout same as those of thereference image.

When photographing is completed, the control unit 21 outputs anotification sound from the speaker 25. The control unit 21 may notifythe similarity between the captured endoscope image 49 and the referenceimage by sound as illustrated in Table 1.

[Table 1]

In a case where photographing of the endoscope image having highsimilarity continues, the notification sound of the major codecontinues, so that the user can proceed with the endoscopic examinationcomfortably. On the other hand, in a case where photographing of anendoscope image having low similarity continues, attention of the usercan be attracted by the minor code and a single sound.

Note that the notification sounds shown in Table 1 are examples. In acase where the similarity is high, a sound emphasizing a sense ofsuccess, such as a sound of “pin-pong” or a melody preferred by theuser, may be used as the notification sound. In a case where thesimilarity is not high, a sound expressing progress, such as “beep” or“honk”, may be used. It is desirable that the user can appropriately setthe presence or absence and the volume of the notification sound.

FIG. 14 is a modification of a screen displayed in a case where the nextregion to be photographed does not appear in the endoscope image 49 andthe guide frame 75 is not displayed. The control unit 21 displays aguide-in-progress mark 77 in which thick arrows are disposed along theouter periphery of the endoscope image field 73. The total number ofthick arrows indicates the total number of still images to bephotographed. The black thick arrow indicates the number of capturedstill images. In the example illustrated in FIG. 14, 3 of 18 images havebeen captured.

Since the guide-in-progress mark 77 is displayed in addition to theprogress display field 71 and the state display field 72, the user caneasily confirm that the mode is a mode in which the guide frame 75 is tobe displayed while being conscious of the progress of the examination.In a case where the guide frame 75 is not displayed, the control unit 21may display the guide-in-progress mark 77 on a steady basis or maydisplay the guide-in-progress mark 77 intermittently.

Since the guide-in-progress mark 77 is displayed in addition to thestate display field 72, the user can easily confirm that the mode is amode in which the guide frame 75 is to be displayed.

FIG. 15 is an example of a screen displayed in a case where the userperforms observation deviating from the guidelines. In FIG. 15, a lesionregion is indicated by grid-like hatching. For example, in a case wherethe user starts special light observation, in a case where the userstarts enlargement observation using a zoom function, in a case wherethe user scatters a pigment, or the like, the control unit 21 determinesthat observation deviating from the guidelines is started.

The control unit 21 displays “Guide Mode Pause” in the state displayfield 72, which means a mode for stopping the display of the guide frame75. The control unit 21 stops display of the guide frame 75. As aresult, the user can observe the entire endoscope image 49 without beingobstructed by the guide frame 75.

FIG. 16 illustrates an example of a screen displayed when the controlunit 21 detects a lesion. Software or the like for detecting a lesionsuch as a cancer or a polyp in real time from the endoscope image 49 hasalready been used, and thus a detailed description thereof will beomitted.

When a lesion is detected, the control unit 21 stops displaying theguide frame 75 and displays a lesion frame 753 surrounding the lesion.The control unit displays a detection notification field 763 fornotifying that a lesion has been detected. The control unit 21 displays“Guide Mode Pause” in the state display field 72, which means a mode forstopping the display of the guide frame 75. The control unit 21 mayoutput the notification sound from speaker 25. The user observes thedetected lesion and performs appropriate treatment based on professionaldetermination.

In a case where the screen described with reference to FIG. 15 or 16 isdisplayed, the control unit 21 receives an instruction to resume themode for displaying the guide frame 75 from the user. For example, in acase where the user sufficiently observes the lesion and determines tocontinue the observation according to the guidelines, the user gives aninstruction to change to a mode for displaying the guide frame 75 by anoperation of the keyboard 17, voice input, or the like. The mode changeinstruction may be assigned to the switch button 36 different from theimage capturing button.

FIG. 17 is a flowchart for explaining the processing flow of a program.The control unit 21 determines whether a lesion is detected (step S501).For example, in a case where an operation such as special lightobservation or enlargement observation is received, the control unit 21determines that a lesion is detected.

The control unit 21 may determine whether a lesion is detected based onthe endoscope image 49. For example, in a case where the color of theendoscope image 49 suddenly changes due to pigment spraying, in a casewhere the distal end of the treatment instrument appears in theendoscope image 49, or the like, the control unit 21 may determine thata lesion is detected. The control unit 21 may receive informationindicating that a lesion has been detected in the endoscope image 49from the lesion detection software.

The control unit 21 may determine that a lesion is detected when atreatment instrument is inserted into the channel. For example, thecontrol unit 21 can detect insertion of a treatment instrument via asensor provided in the channel inlet 37.

The control unit 21 may determine that a lesion is detected based on aninstruction from the user received through the switch button 36, voiceinput, or the like. In a case where the user desires to observe theendoscope image 49 without being obstructed by the guide frame 75, theuser can shift the mode to a mode in which the guide frame 75 is notdisplayed. In step S501, the control unit 21 realizes a function of alesion detection unit that detects that a lesion is being observed.

When it is determined that no lesion is detected (NO in step S501), thecontrol unit 21 inputs the endoscope image 49 to the first learningmodel 61 and acquires a region to be photographed (step S502). Thecontrol unit 21 determines whether there is a region to be photographedwhere the dimension of the detection frame 653 acquired from the firstlearning model 61 is larger than a predetermined value and a probabilityof exceeding a predetermined threshold value is output (step S503). Whenit is determined that there is no region to be photographed (NO in stepS503), the control unit 21 returns to step S501.

When it is determined that there is the region to be photographed (YESin step S503), the control unit 21 activates a subroutine of targetselection (step S504). The subroutine of the target selection is asubroutine of selecting the region to be photographed for displaying theguide frame 75 from the region to be photographed output from the firstlearning model 61. The return value of the subroutine of targetselection includes a variable indicating whether the selection of theregion to be photographed is successful. The processing flow of thesubroutine of target selection will be described later.

The control unit 21 determines whether the subroutine of targetselection has succeeded in selecting the region to be photographed (stepS505). When it is determined that it is not successful (NO in stepS505), the control unit 21 returns to step S501.

When it is determined that it is successful (YES in step S505), thecontrol unit 21 starts the subroutine of the guide display (step S506).The subroutine of the guide display is a subroutine that displays theguide frame 75 superimposed on the endoscope image 49. The processingflow of the subroutine of the guide display will be described later.

The control unit 21 performs tracking of changing the display of theguide frame 75 in accordance with the change in the endoscope image 49(step S507). The control unit 21 determines whether the endoscope image49 is in an appropriate state for photographing the region to bephotographed (step S508). Specifically, in a case where most of the areaof the endoscope image 49, for example, 90% or more, is surrounded bythe guide frame 75 being tracked, the control unit 21 determines that itis in an appropriate state.

When it is determined that it is in an appropriate state (YES in stepS508), the control unit 21 activates a subroutine of a record (stepS509). The subroutine of a record is a subroutine for recording theendoscope image 49 in the examination DB 67. The processing flow of thesubroutine of a record will be described later.

The control unit 21 determines whether to end the process (step S510).For example, in a case where all the regions to be photographed definedin the guidelines have been photographed or in a case where theinsertion portion 31 has been removed from the medical examinee, thecontrol unit 21 determines that the process ends. When it is determinedthat the process ends (YES in step S510), the control unit 21 ends theprocess.

When it is determined that the process does not end (NO in step S510) orwhen it is determined that the proper state is not obtained even afterthe predetermined time elapses (NO in step S508), the control unit 21returns to step S501.

When it is determined that a lesion is detected (YES in step S501), thecontrol unit 21 shifts the mode to a mode in which the guide frame 75 isnot displayed as described with reference to FIG. 9 or 10 (step S521).The control unit 21 waits until receiving an instruction to resume theoperation in the mode for displaying the guide frame 75 (step S522).After receiving the instruction, the control unit 21 returns to stepS501 and resumes the operation of the mode for displaying the guideframe 75.

FIG. 18 is a flowchart for explaining a processing flow of a subroutineof target selection. The subroutine of the target selection is asubroutine of selecting the region to be photographed for displaying theguide frame 75 from the region to be photographed output from the firstlearning model 61.

The control unit 21 determines whether a next region to be photographedconforming to a predetermined photographing sequence is included in theregion to be photographed output from the first learning model 61 (stepS531). When determining that the region to be photographed is included(YES in step S531), the control unit 21 selects the region to bephotographed as the display target of the guide frame 75 (step S532).The control unit 21 determines that the processing result of thesubroutine of target selection is success (step S533).

When determining that the region to be photographed is not included (NOin step S531), the control unit 21 determines that the processing resultof the subroutine of target selection is failure (step S534).

FIG. 19 is a flowchart for explaining a processing flow of a subroutineof the guide display. The subroutine of the guide display is asubroutine that displays the guide frame 75 superimposed on theendoscope image 49.

In the control unit 21, the control unit 21 matches the endoscope image49 captured in real time with the reference image acquired from thereference image DB 66, and extracts a portion corresponding to thereference image in the endoscope image 49 (step S551).

The control unit 21 determines whether the region to be photographed istoo close to the distal end portion 313 (step S552). For example, in acase where the entire endoscope image 49 corresponds to a portion of thereference image, that is, in a case where the reference image is placedbeyond the endoscope image 49, the control unit 21 determines that theregion to be photographed is too close to the distal end portion 313.

When it is determined that it is too close (YES in step S552), thecontrol unit 21 superimposes the approach index 751 on the endoscopeimage 49 as described with reference to FIG. 9 and displays it in theendoscope image field 73 (step S553). When it is determined that it isnot too close (NO in step S552), the control unit 21 determines whetherrotation of the insertion portion 31 is necessary (step S554). Forexample, in a case where the edge of the matched reference image isrotated with respect to the endoscope image 49, the control unit 21determines that rotation is necessary.

When determining that the rotation is necessary (YES in step S554), thecontrol unit 21 superimposes the rotation index 752 on the endoscopeimage 49 as described with reference to FIG. 11 and displays it in theendoscope image field 73 (step S555). When it is determined that therotation is unnecessary (NO in step S554), the control unit 21superimposes the guide frame 75 on the endoscope image 49 and displaysit in the endoscope image field 73 (step S556). After that, the controlunit 21 ends the process.

In step S556, the control unit 21 uses the edge of the range matchedwith the reference image in step S551 for the guide frame 75.

In step S556, the control unit 21 may change the color, thickness, orline type of the guide frame 75 based on the probability acquired fromthe first learning model 61. The user can determine the degree to whichthe guide frame 75 can be trusted.

When determining that it is not too close (NO in step S552), the controlunit 21 may proceed to step S556 without determining whether therotation is necessary. In this case, for example, as described withreference to FIG. 10, the rotation of the endoscope image 49 and thereference image is expressed by the guide frame 75. That is, guide frame75 also serves as the rotation index 752.

When the dimension of the detection frame 653 acquired from the firstlearning model 61 is smaller than the predetermined dimension, thecontrol unit 21 may use the detection frame 653 as the guide frame 75instead of executing the subroutine of the guide display. When thedistal end portion 313 is far from the observation target region and theguide frame 75 is displayed to be small on the screen, the load on thecontrol unit 21 can be reduced by using the detection frame 653.

The user operates the endoscope 30 by referring to the guide frame 75,the approach index 751, and the rotation index 752 displayed with themsuperimposed on the endoscope image 49 by the subroutine of the guidedisplay, so that the endoscope image 49 having high similarity with thereference image can be photographed. That is, the control unit 21functions as a guide output unit that displays a guide for increasingthe similarity between the endoscope image 49 and the reference image bya subroutine of the guide display.

FIG. 20 is a flowchart for explaining a processing flow of a subroutineof a record. The subroutine of a record is a subroutine for recordingthe endoscope image 49 in the examination DB 67.

The control unit 21 inputs the endoscope image 49 to the second learningmodel 62 and acquires a region in which the foreign matter such as mucusappears (step S561). The control unit 21 determines whether there is aregion in which a probability exceeding the predetermined thresholdvalue is output (step S562). In a case where it is determined that thereis a region in which the foreign matter appears with a probabilityexceeding the threshold value (YES in step S562), the control unit 21displays the screen described with reference to FIG. 12 and notifies theuser of the screen (step S563).

The user who has confirmed the notification removes the foreign matterusing the sub water supply function, water supply from the channel, orthe like. When it is determined that the image cannot be removed or theimage does not need to be removed, the user presses the image capturingbutton to instruct image capturing.

The control unit 21 determines whether the photographing instruction hasbeen received (step S564). When it is determined that the photographinginstruction has not been received (NO in step S564), the control unit 21returns to step S561.

When it is determined that there is no region in which the foreignmatter appears with a probability exceeding the threshold value (NO instep S562), or when it is determined that the photographing instructionis received (YES in step S564), the control unit 21 displays thephotograph-in-progress index 761 described using FIG. 13 to notify theuser that the endoscope image 49 is to be photographed. The control unit21 may output sound from the speaker 25 to make notification to theuser. The user stops the distal end portion 313 to prevent imageblurring. The control unit 21 calculates similarity between theendoscope image 49 and the reference image (step S565).

The control unit 21 stores the endoscope image 49 in the examination DB67 (step S566). Specifically, the control unit 21 creates a new recordin the examination DB 67. The control unit 21 records the doctor ID, themedical examinee ID, and the date in the doctor ID field, the medicalexaminee ID field, and the date field, respectively. The control unit 21records numbers in the number field by a serial number. The control unit21 records the name of the region to be photographed in the region namefield. The control unit 21 records the similarity calculated in stepS565 in the similarity field. The control unit 21 records the endoscopeimage 49 in the image field.

The control unit 21 displays in the progress display field 71 thatphotographing of one region to be photographed is finished. For example,when using the screen described with reference to FIG. 7, the controlunit 21 creates a thumbnail of the image recorded in the examination DB67 and displays the thumbnail in the progress display field 71. Whenusing the screen described with reference to FIG. 8, the control unit 21displays a check mark in a check box corresponding to the image recordedin the examination DB 67. After that, the control unit 21 ends theprocess.

In a case where the release operation for acquiring a still image is notperformed in step S563, and the endoscope image 49 input to the secondlearning model 62 is recorded in the examination DB 67 in step S561, thesimilarity may be calculated simultaneously with the foreign matterdetection in step S561, and step S565 may be omitted.

Note that, in a case where the mode is shifted to a mode in which theguide frame 75 is not displayed in step S521 of the program describedwith reference to FIG. 17, or in a case where the user presses the imagecapturing button at timing other than step S566, the control unit 21records the endoscope image 49 in the examination DB 67 as in step S564.In such a case, the control unit 21 records “-” in the region name fieldand the similarity field.

According to the present embodiment, it is possible to provide theendoscope system 10 that records the endoscope image 49 for the sameportion as the reference image defined in the guidelines with the samemagnification and layout. The endoscope system 10 of the presentembodiment can assist prevention of overlooking in endoscopicexamination and improvement of efficiency of double check.

According to the present embodiment, since the guide frame 75 isdisplayed in accordance with the photographing sequence defined in theguidelines, it is possible to provide the endoscope system 10 thatassists photographing in the order according to the guidelines. It ispossible to shorten the time required for endoscopic examination byhabituating photographing in the order according to the guidelines usingthe endoscope system 10 according to the present embodiment.

It is possible to provide the endoscope system 10 that does not hinderthe work of the doctor by stopping the display of the guide frame 75 ina case where the doctor finds a lesion and starts detailed observationsuch as special light observation.

It is possible to provide the endoscope system 10 that preventsautomatic photographing of the endoscope image 49 in which the foreignmatter that hinders diagnosis appears by making notification to the userwithout capturing an image in a case where the foreign matter such asmucus is detected in the endoscope image 49. Even in a case where theforeign matter is detected, when the user presses the image capturingbutton, the endoscope image 49 is photographed, so that it is possibleto provide the endoscope system 10 that operates in deference to thedetermination of the doctor.

Second Embodiment

The present embodiment relates to an endoscope system 10 that displays aplurality of guide frames 75 and an endoscope image 49 with the framessuperimposed in the image. Descriptions regarding common portions withthe first embodiment will be omitted.

FIGS. 21 and 22 are explanatory diagrams for explaining screen displayexamples of the second embodiment. In the screen illustrated in FIG. 21,a plurality of guide frames 75 is displayed in the endoscope image field73. Each guide frame 75 is indicated by a broken line, a one-dot chainline, and a two-dot chain line.

Similarly to the screen described with reference to FIG. 8, the controlunit 21 displays a list of the names of the regions to be photographeddefined in the guidelines in the progress display field 71. The controlunit 21 displays a check box 713 on the right side of each name. Achecked check box 713 means that the image has been captured.

The control unit 21 surrounds the names of the regions to bephotographed corresponding to the guide frames 75 indicated by thebroken line, the one-dot chain line, and the two-dot chain line with theframe indexes 714 indicated by the broken line, the one-dot chain line,and the two-dot chain line, respectively. Note that the associationbetween the guide frame 75 and the region to be photographed may beindicated by, for example, the color of the guide frame 75 and thecharacter color of the region to be photographed or the color of theline surrounding the region to be photographed.

FIG. 22 is an example of a screen displayed by the control unit 21 afterphotographing of “lesser curvature of antrum” indicated by the guideframe 75 of the two-dot chain line is completed. The control unit 21displays a check mark in the check box 713 of “lesser curvature ofantrum”. After the “lesser curvature of antrum” is photographed, theposition of the distal end portion 313 is returned to the same positionas when the user captures the image of FIG. 21.

The control unit 21 does not display the guide frame 75 and the frameindex 714 corresponding to the photographed “lesser curvature ofantrum”. The user viewing the screen illustrated in FIG. 22 operates theendoscope 30 so as to photograph the “greater curvature of antrum” orthe “anterior wall of antrum” whose guide frame 75 is displayed.

FIG. 23 is a flowchart for explaining a processing flow of a subroutineof target selection of the second embodiment. The subroutine describedwith reference to FIG. 23 is a subroutine used instead of the subroutinedescribed with reference to FIG. 18.

The control unit 21 deletes a region having a probability less than thepredetermined threshold value among the regions to be photographedoutput from the first learning model 61 (step S571). The control unit 21deletes the already photographed region from the regions to bephotographed remaining in step S571 (step S572).

The control unit 21 determines whether there is a region to bephotographed remaining after step S571 (step S573). When it isdetermined that there is a remaining region to be photographed (YES instep S533), the control unit 21 selects the remaining region to bephotographed as a display target of the guide frame 75 (step S574).

The control unit 21 determines that the processing result of thesubroutine of target selection is successful (step S575). When it isdetermined that there is no remaining region to be photographed (NO instep S573), the control unit 21 determines that the processing result ofthe subroutine of target selection is failure (step S576). After stepS575 or step S576 ends, the control unit 21 ends the process.

FIG. 24 is a flowchart for explaining a processing flow of a subroutineof the guide display of the second embodiment. The subroutine describedwith reference to FIG. 24 is a subroutine used instead of the subroutinedescribed with reference to FIG. 19.

The control unit 21 determines whether the user can specify the regionto be photographed to be photographed next among the region to bephotographed determined to be the display target of the guide frame 75in the subroutine of target selection (step S581).

Specifically, the control unit 21 determines whether there is a regionto be photographed in which the center of the detection frame 653acquired from the first learning model 61 substantially matches thecenter of the endoscope image 49 and the dimension of the detectionframe 653 is larger than a predetermined value. This is because theregion to be photographed satisfying such a condition is the region tobe photographed to which the user brings the distal end portion 313close in front view.

When determining that the region to be photographed can be specified(YES in step S581), the control unit 21 displays the name of the nextregion to be photographed by surrounding it by the frame index 714. Thecontrol unit 21 matches the endoscope image 49 photographed in real timewith the reference image of the region to be photographed specified tobe photographed next, and extracts a portion corresponding to thereference image in the endoscope image 49 (step S591).

The control unit 21 determines whether the next region to bephotographed is too close to the distal end portion 313 (step S592). Forexample, in a case where the entire endoscope image 49 corresponds to aportion of the reference image, that is, in a case where the referenceimage is placed beyond the endoscope image 49, the control unit 21determines that the region to be photographed is too close to the distalend portion 313.

When it is determined that it is too close (YES in step S592), thecontrol unit 21 superimposes the approach index 751 on the endoscopeimage 49 as described with reference to FIG. 9 and displays it in theendoscope image field 73 (step S593). When it is determined that it isnot too close (NO in step S592), the control unit 21 determines whetherrotation of the insertion portion 31 is necessary (step S594).

When determining that the rotation is necessary (YES in step S594), thecontrol unit 21 superimposes the rotation index 752 on the endoscopeimage 49 as described with reference to FIG. 11 and displays it in theendoscope image field 73 (step S595). When it is determined that therotation is unnecessary (NO in step S594), the control unit 21superimposes the guide frame 75 corresponding to the target region to bephotographed next on the endoscope image 49 and displays it in theendoscope image field 73 (step S596).

After completion of step S593, step S595, or step S596, the control unit21 determines whether there is a region to be photographed fordisplaying the guide frame 75 (step S601). When it is determined thatthere is no region to be photographed (NO in step S601), the controlunit 21 ends the process. Note that, in a case where it is determinedthat there is no region to be photographed, in step S507 after returningfrom this subroutine to the program described with reference to FIG. 17,there is no guide frame to be tracked, so that it is determined that itis not appropriate state (NO in step S508).

When it is determined that there is the region to be photographed (YESin step S601), or when it is determined that the user cannot specify theregion to be photographed to be photographed next (NO in step S581), thecontrol unit 21 selects the region to be photographed for displaying theguide frame 75 next (step S602). The control unit 21 displays the nameof the selected region to be photographed by surrounding it by the frameindex 714.

The control unit 21 matches the endoscope image 49 captured in real timewith the reference image of the region to be photographed selected instep S602, and extracts a portion corresponding to the reference imagein the endoscope image 49 (step S603).

The control unit 21 displays the guide frame 75 and the endoscope image49 with the frame superimposed on the image based on the matching result(step S604). The control unit 21 determines whether display of all guideframes 75 has been finished (step S605). When it is determined that itis not finished (NO in step S605), the control unit 21 returns to stepS602. When it is determined that it is finished (YES in step S605), thecontrol unit 21 ends the process.

Note that the guide frame 75 displayed in step S596 may be displayed ina conspicuous form such as a thicker line or a darker color than that ofthe guide frame 75 displayed in step S604. It is possible to provide theendoscope system 10 in which the user can easily confirm the region tobe photographed determined to be photographed next by the control unit21.

According to the present embodiment, it is possible to provide theendoscope system 10 that assists photographing of a region to bephotographed appropriately selected by a user in a case where aplurality of regions to be photographed is included in the sameendoscope image 49.

According to the present embodiment, since the guide frames indicatingthe unphotographed regions to be photographed among the plurality ofregions to be photographed included in the endoscope image 49 aredisplayed at the same time, the user can appropriately performphotographing in the order of easily photographing. Since the guideframe 75 indicating the photographed target region is not displayed,overlapping photographing can be prevented. Note that, in a case wherethe user wants to photograph the same region to be photographed again,the photographing can be performed by operating the image capturingbutton.

As described above, it is possible to provide the endoscope system 10that prevents photographing omission and overlapping photographing. Inaddition, since the user can perform photographing in an arbitraryorder, comfortable operation can be assisted without hindering theoperation of the user.

Third Embodiment

The present embodiment relates to the endoscope system 10 that displaysa notification when a user performs photographing in an order differentfrom the order of the region to be photographed defined in theguidelines. Description of portions different from those in the secondembodiment is omitted.

FIGS. 25 and 26 are explanatory diagrams for explaining the screendisplay examples of the third embodiment. FIG. 25 illustrates an exampleof a screen displayed by the control unit 21 when the antrum is observedwithout photographing the “esophagogastric junction”. The control unit21 surrounds the row of the “esophagogastric junction” with anotification index 715 and notifies that the photographing has not beenperformed. That is, by displaying the notification index 715, thecontrol unit 21 functions as a second notification output unit thatoutputs a notification in a case where only the guide frame 75 that doesnot conform to the predetermined observation sequence defined in theguidelines is displayed.

Note that the control unit 21 may make the row of the “esophagogastricjunction” conspicuous by means of blinking or displaying in red, forexample. The control unit 21 may further output a voice from speaker 25to call user's attention. The voice may be a language such as “theesophagogastric junction is not photographed” or a notification soundexpressing a sense of caution such as “beep”. The control unit 21 alsofunctions as a second notification output unit by these operations.

When the user simply forgets to photograph the esophagogastric junction,the user appropriately operates the endoscope 30 to direct the distalend portion 313 toward the vicinity of the esophagogastric junction. Thecontrol unit 21 displays the guide frame 75 corresponding to theesophagogastric junction to assist the user.

When an image corresponding to the reference image cannot be capturedfor the esophagogastric junction due to the presence of a lesion or thelike, the user instructs to cancel the notification by operating theswitch button 36, voice input, or the like. FIG. 26 illustrates a statein which the notification related to the esophagogastric junction iscanceled. The control unit 21 displays a strike-through line in thecheck box 713 of the esophagogastric junction and deletes thenotification index 715. Thereafter, the user can continue the endoscopicexamination without being bothered by the notification.

FIG. 27 is a flowchart for explaining a processing flow of a subroutineof the target selection of the third embodiment. The subroutinedescribed with reference to FIG. 27 is a subroutine used instead of thesubroutine described with reference to FIG. 23.

The control unit 21 deletes a region having a probability less than thepredetermined threshold value among the regions to be photographedoutput from the first learning model 61 (step S571). The control unit 21deletes the already photographed region from the regions to bephotographed remaining in step S571 (step S572).

The control unit 21 determines whether there is a region to bephotographed remaining after step S571 (step S573). When it isdetermined that there is no remaining region to be photographed (NO instep S573), the control unit 21 determines that the processing result ofthe subroutine of target selection is failure (step S576).

When it is determined that there is remaining regions to be photographed(YES in step S533), the control unit 21 determines whether there is anunphotographed region set in the guidelines that photographing should beperformed before any of the remaining regions to be photographed (stepS611). When it is determined that there is an unphotographed region (YESin step S611), the control unit 21 displays the notification index 715described with reference to FIG. 25 and makes notification to the user.

The control unit 21 determines whether an instruction by the user tostop the notification of the unphotographed region has been received(step S613). When it is determined that the instruction has beenreceived (YES in step S613), the control unit 21 deletes thenotification index 715 and stops the notification (step S614). Thecontrol unit 21 temporarily stores, in the main storage device 22 or theauxiliary storage device 23, that the instruction to stop thenotification has been received, and does not make the notificationregarding the unphotographed region in step S612 to be executed in aloop thereafter.

When it is determined that there is no unphotographed region (NO in stepS611), when it is determined that an instruction to stop thenotification is not received (NO in step S613), or after the end of stepS614, the control unit 21 selects all remaining regions to bephotographed as display targets of the guide frame 75 (step S574).

The control unit 21 determines that the processing result of thesubroutine of target selection is successful (step S575). After stepS575 or step S576 ends, the control unit 21 ends the process.

According to the present embodiment, it is possible to provide theendoscope system 10 that attracts the attention of the user in a casewhere the user photographs the next region to be photographed withoutphotographing the region to be photographed. It is possible to providethe endoscope system 10 that does not trouble the user more thannecessary by accepting the stop of the notification in a case where theuser intentionally omits photographing of the region to be photographed.

Fourth Embodiment

The present embodiment relates to the endoscope system 10 that displayscautions at the time of photographing Descriptions regarding commonportions with the first embodiment will be omitted.

FIG. 28 is an explanatory diagram for explaining a screen displayexample of the fourth embodiment. In a case where the guide frame 75 isdisplayed at the center of the endoscope image 49 with a size equal toor larger than a predetermined size, it can be estimated that the userhas started preparation for photographing. The control unit 21 displaysthe reference image corresponding to the guide frame 75 in a referenceimage field 78, and displays the attention item at the time ofphotographing the region to be photographed in an attention item field781 below the reference image field. Note that the attention item isrecorded in association with the reference image in the reference imageDB 66, for example.

According to the present embodiment, the user can perform cleaning orthe like according to the attention item while performing the operationof directing the distal end portion 313 toward the region to bephotographed. Therefore, it is possible to provide the endoscope system10 that assists the user so that an appropriate image can be smoothlycaptured.

Fifth Embodiment

The present embodiment relates to the endoscope system 10 in which thecontrol unit 21 automatically controls the bending section 312 tocapture an image close to the reference image. Descriptions regardingcommon portions with the first embodiment will be omitted. FIG. 29 is anexplanatory diagram for explaining the configuration of the endoscopesystem 10 of the fifth embodiment.

In FIG. 29, a configuration related to the operation of the bendingsection 312 is schematically described, and description of otherportions is omitted. In FIG. 29, one bending knob 35 is schematicallyillustrated. By having the same configuration in both the U-D (Up-Down)direction and the R-L (Right-Left) direction, it is possible toautomatically control so-called four-direction bending.

A sensor 355 that detects an operation amount by a user is attached tothe bending knob 35. The sensor 355 is connected to the processor for anendoscope 20. A bending wire 351 that bends the bending section 312 isconnected to an actuator 356 provided in the operation unit 32.

When the user operates the bending knob 35, the control unit 21 controlsthe actuator 356 according to the operation amount detected by thesensor 355. The bending section 312 is bent by pulling the bending wire351 by the actuator 356. Therefore, the user can perform the bendingoperation of the bending section 312 by operating the bending knob 35similarly to the normal endoscope 30.

FIG. 30 is an explanatory diagram for explaining a screen displayexample of the fifth embodiment. In the present embodiment, when theguide frame 75 approaches the edge of the endoscope image 49, thecontrol unit 21 shifts to the automatic control mode and displays anautomatic control-in-progress mark 764. That is, by displaying theautomatic control-in-progress mark 764, the control unit 21 functions asa third notification output unit that outputs a notification in a casewhere the endoscope 30 is automatically controlled.

In the automatic control mode, the control unit 21 automaticallycontrols the bending section 312 via the actuator 356 so that the guideframe 75 matches the edge of the endoscope image 49, performs alignmentin an appropriate state, and then automatically performs photographing.

FIG. 31 is a flowchart for explaining a processing flow of the programof the fifth embodiment. Since the processing flow from the start tostep S507 is similar to the flowchart described with reference to FIG.17, the description thereof will be omitted.

The control unit 21 determines whether the endoscope image 49 is in thevicinity of an appropriate state for photographing a region to bephotographed (step S621). Specifically, in a case where the edge of theendoscope image 49 and the guide frame 75 being tracked are close toeach other, the control unit 21 determines that they are in the vicinityof an appropriate state. In a case where the similarity between theendoscope image 49 and the reference image exceeds a predeterminedthreshold value, the control unit 21 may determine that they are in thevicinity of the appropriate state.

When it is determined that they are in the vicinity of the appropriatestate (YES in step S621), the control unit 21 shifts to the automaticcontrol mode and automatically controls the actuator 356 so that theendoscope image 49 approaches the reference image (step S622).

The control unit 21 determines whether the state of the endoscope image49 has deteriorated by the automatic control (step S623). Specifically,in a case where the edge of the endoscope image 49 and the guide frame75 being tracked are away from each other, the control unit 21determines that the state has deteriorated. The control unit 21 maydetermine that the state has deteriorated in a case where the similaritybetween the endoscope image 49 and the reference image has decreased.

When it is determined that the state has deteriorated (YES in stepS621), the control unit 21 returns to step S501. When it is determinedthat the state has not deteriorated (NO in step S621), the control unit21 determines whether the endoscope image 49 is in an appropriate statefor photographing the region to be photographed (step S624).Specifically, in a case where most of the area of the endoscope image49, for example, 90% or more, is surrounded by the guide frame 75 beingtracked, the control unit 21 determines that it is in an appropriatestate.

When it is determined that it is not in an appropriate state (NO in stepS624), the control unit 21 returns to step S622. When it is determinedthat it is in an appropriate state (YES in step S624), the control unit21 activates a subroutine of a record (step S509). Since the subsequentprocessing is the same as the program described with reference to FIG.17, description is omitted.

According to the present embodiment, since fine adjustment immediatelybefore photographing is automatically performed, it is possible toprovide the endoscope system 10 with less burden on the user. Since theoperation other than the fine adjustment is manually performed by theuser, it is possible to provide the endoscope system 10 capable ofperforming the endoscopic examination similarly to the related art.

Note that the endoscope system 10 may be configured such that thecontrol unit 21 can automatically control air/water supply, sub watersupply, adjustment of zoom magnification, and the like in addition tothe bending section 312. By performing the automatic control, thecontrol unit 21 functions as an automatic control unit that operates theendoscope 30 so as to increase the similarity between the endoscopeimage 49 and the reference image.

Sixth Embodiment

The present embodiment relates to an endoscope system 10 having atraining function of an endoscopic examination manipulation according tothe guidelines. Descriptions regarding common portions with the firstembodiment will be omitted.

FIG. 32 is an explanatory diagram for explaining a screen displayexample of the sixth embodiment. After completion of the endoscopicexamination according to the guidelines, the control unit 21 displaysthe screen illustrated in FIG. 32. A score field 79 is displayed on thescreen illustrated in FIG. 32. In the score field 79, the number ofimages captured according to the guidelines in a series of endoscopicexaminations and representative values of similarity such as an averagepoint, a highest point, and a lowest point of similarity are displayed.The control unit 21 may display the required time and the like.

In a case where the examination of one medical examinee ends and theendoscope 30 is removed from the medical examinee, the control unit 21calculates the score based on the examination DB 67 and displays thescreen illustrated in FIG. 32. It can be seen that the higher thesimilarity is, the more appropriately the endoscope image 49 accordingto the guidelines can be captured, and the higher the proficiency levelis. That is, the control unit 21 functions as an evaluation display unitthat displays the evaluation of the manipulation by the user based onthe similarity by outputting the screen illustrated in FIG. 32.

The control unit 21 may display the transition of the score in apredetermined period such as one month or one year as a graph. The usercan check the degree of improvement of the endoscopic examination byleveling the difficulty level of insertion for each subject by thetransition of the score.

The control unit 21 may display the comparison with the score of anotheruser by, for example, a deviation value, a graph, or the like. The usercan grasp the relative level of his or her manipulation.

According to the present embodiment, by displaying the score of theendoscopic examination manipulation, it is possible to provide theendoscope system 10 that assists improvement of the manipulation by theuser, that is, training of the user. The user can use the endoscopesystem 10 of the present embodiment as a training device that performstraining of a manipulation of endoscopic examination.

Seventh Embodiment

The present embodiment relates to an endoscope system 10 used for afollow-up observation of a patient. Descriptions regarding commonportions with the first embodiment will be omitted.

For example, after the endoscopic treatment is performed, the same placeis periodically observed, and a follow-up observation is performed toconfirm progress of healing, presence or absence of recurrence, or thelike. In the follow-up observation, it is desirable to photograph animage with the same magnification and layout as possible every time.

FIG. 33 is an explanatory diagram for explaining a record layout of thefollow-up observation DB. The follow-up observation DB includes amedical examinee ID field, a photographing date field, and an imagefield.

In the medical examinee ID field, the medical examinee ID is recorded.In the photographing date field, a photographing date is recorded. Inthe image field, a file of still images is recorded.

The follow-up observation DB has one record for each follow-upobservation. The follow-up observation DB is stored in the auxiliarystorage device 23 or an external mass storage device connected to theprocessor for an endoscope 20.

FIG. 34 is a flowchart for explaining a processing flow of the programof the seventh embodiment. The control unit 21 receives an input of themedical examinee ID (step S631). The control unit 21 searches thefollow-up observation DB using the medical examinee ID as a key,extracts a record to acquire a reference image from the image field(step S632). The reference image is, for example, an image at the timeof the previous follow-up observation. An image at the time of the firstfollow-up observation may be used as the reference image.

The control unit 21 acquires the real-time endoscope image 49 (stepS633). The control unit 21 calculates similarity between the endoscopeimage 49 and the reference image (step S634). The control unit 21determines whether the calculated similarity is higher than apredetermined threshold value (step S635).

When it is determined that the similarity is higher than thepredetermined threshold value (YES in step S635), the control unit 21activates a subroutine of the guide display (step S636). The subroutineof the guide display is the same subroutine as the subroutine describedwith reference to FIG. 19.

The control unit 21 performs tracking of changing the display of theguide frame 75 in accordance with the change in the endoscope image 49(step S637). The control unit 21 determines whether the endoscope image49 is in an appropriate state for photographing an image of thefollow-up observation (step S538). Specifically, in a case where most ofthe area of the endoscope image 49, for example, 90% or more, issurrounded by the guide frame 75 being tracked, the control unit 21determines that it is in an appropriate state.

When it is determined that it is in an appropriate state (YES in stepS638), the control unit 21 photographs a new record in the follow-upobservation DB and records the endoscope image 49 (step S639). Afterthat, the control unit 21 ends the process.

When it is determined that the similarity is lower than thepredetermined threshold value (NO in step S635) or when it is determinedthat it is not in an appropriate state (NO in step S638), the processreturns to step S633.

According to the present embodiment, it is possible to provide theendoscope system 10 that repeatedly records the follow-up observationimage with the same image and layout. The doctor can accuratelydetermine the progress such as the state of cure or the state ofrecurrence.

Eighth Embodiment

FIG. 35 is a functional block diagram of the processor for an endoscope20 of the eighth embodiment. The processor for an endoscope 20 includesan image acquisition unit 81, a region acquisition unit 82, and an imageoutput unit 83. The image acquisition unit 81 acquires the endoscopeimage 49. The region acquisition unit 82 inputs the endoscope image 49acquired by the image acquisition unit 81 to the first learning model 61that outputs a target region corresponding to a predetermined region tobe photographed when the endoscope image 49 is input, and acquires thetarget region. The image output unit 83 superimposes the endoscope image49 acquired by the image acquisition unit 81 and the index 75 indicatingthe target region acquired by the region acquisition unit 82 to outputthem.

Ninth Embodiment

The present embodiment relates to a mode for realizing the endoscopesystem 10 of the present embodiment by operating the general-purposecomputer 90 and the program 97 in combination. FIG. 36 is an explanatorydiagram for explaining the configuration of the endoscope system 10 ofthe ninth embodiment. Descriptions regarding common portions with thefirst embodiment will be omitted.

The endoscope system 10 according to the present embodiment includes anendoscope 30, a processor for an endoscope, and a computer 90. Thecomputer 90 includes the control unit 21, the main storage device 22,the auxiliary storage device 23, the communication unit 24, the speaker25, the display device I/F 26, the input device I/F 27, the reading unit28, and the bus. The computer 90 is an information device such as ageneral-purpose personal computer, a tablet, or a server computer.

The computer 90 is connected to the processor for an endoscope 20 in awired or wireless manner to acquire the endoscope image 49 from theprocessor for an endoscope 20 in real time.

A program 97 is recorded in a portable recording medium 96. The controlunit 21 reads the program 97 via the reading unit 28 and stores it inthe auxiliary storage device 23. Further, the control unit 21 may readthe program 97 stored in a semiconductor memory 98 such as a flashmemory mounted in the computer 90. Further, the control unit 21 maydownload the program 97 from the communication unit 24 and anotherserver computer (not illustrated) connected via a network (notillustrated), and store the program 97 in the auxiliary storage device23.

The program 97 is installed as a control program for the computer 90 andis loaded and executed in the main storage device 22. As a result, thecomputer 90 functions as a control device of the endoscope system 10.

Technical features (constitutional requirements) described in therespective embodiments can be combined with each other, and newtechnical features can be formed with the combination.

The embodiments herein are disclosed for purposes of illustration in allrespects and not limitation. The scope of the present invention isdefined not by the above-described meaning but by claims, and intends toinclude all modifications within meaning and a scope equal to claims.

REFERENCE SIGNS LIST

-   10 endoscope system-   16 display device-   17 keyboard-   20 processor for an endoscope-   21 control unit-   22 main storage device-   23 auxiliary storage device-   24 communication unit-   25 speaker-   26 display device I/F-   27 input device I/F-   28 reading unit-   29 endoscope connection unit-   30 endoscope-   31 insertion portion-   311 soft portion-   312 bending section-   313 distal end portion-   32 operation unit-   321 air/water supply button-   322 suction button-   33 universal cord-   34 connector unit-   35 bending knob-   351 bending wire-   355 sensor-   356 actuator-   36 switch button-   37 channel inlet-   371 forceps plug-   49 endoscope image-   61 first learning model-   62 second learning model-   651 region candidate extraction unit-   652 classification unit-   653 detection frame-   66 reference image DB-   67 examination DB-   71 progress display field-   713 check box-   714 frame index-   715 notification index-   72 state display field-   73 endoscope image field-   74 past image field-   75 guide frame (index)-   751 approach index-   752 rotation index-   753 lesion frame-   754 accretion frame-   761 photograph-in-progress index-   762 accretion mark-   763 detection notification field-   764 automatic control-in-progress mark-   77 guide-in-progress mark-   78 reference image field-   781 attention item field-   79 score field-   81 image acquisition unit-   82 region acquisition unit-   83 image output unit-   90 computer-   96 portable recording medium-   97 program-   98 semiconductor memory

1. A processor for an endoscope comprising: an image acquisition unitthat acquires an endoscope image; a region acquisition unit that inputsthe endoscope image acquired by the image acquisition unit to a firstlearning model that outputs a target region corresponding to apredetermined region to be photographed when the endoscope image isinput to acquire the target region; and an image output unit thatoutputs the endoscope image acquired by the image acquisition unit andan index indicating the target region acquired by the region acquisitionunit with the endoscope image and the index superimposed.
 2. Theprocessor for an endoscope according to claim 1, further comprising: aguide output unit that outputs a guide for increasing similarity betweenthe endoscope image acquired by the image acquisition unit and areference image associated with a region to be photographed with theguide superimposed on the endoscope image.
 3. The processor for anendoscope according to claim 1, further comprising: an image recordingunit that records an endoscope image acquired by the image acquisitionunit in a case where it is determined that similarity between theendoscope image acquired by the image acquisition unit and a referenceimage associated with a region to be photographed is higher than apredetermined value.
 4. The processor for an endoscope according toclaim 3, further comprising: a first notification output unit thatoutputs a notification when the image recording unit records anendoscope image.
 5. The processor for an endoscope according to claim 4,wherein the first notification output unit outputs a notificationaccording to similarity between a recorded endoscope image and areference image.
 6. The processor for an endoscope according to claim 3,wherein the image recording unit records an endoscope image acquired bythe image acquisition unit when receiving an instruction from a user. 7.The processor for an endoscope according to claim 1, wherein the firstlearning model outputs a plurality of target regions corresponding to aplurality of respective regions to be photographed when an endoscopeimage is input, and the image output unit outputs an endoscope image andan index indicating each of a plurality of target regions with theendoscope image and the index superimposed with each other.
 8. Theprocessor for an endoscope according to claim 7, wherein the imageoutput unit does not superimpose an index corresponding to a region tobe photographed for which a corresponding endoscope image has beenrecorded.
 9. The processor for an endoscope according to claim 1,further comprising: a second notification output unit that outputs anotification when the image output unit displays only an indexcorresponding to a reference image that does not conform to apredetermined observation sequence.
 10. The processor for an endoscopeaccording to claim 1, further comprising: a lesion detection unit thatdetects that a lesion is being observed, wherein the region acquisitionunit stops an operation in a case where the lesion detection unitdetects that a lesion is being observed.
 11. The processor for anendoscope according to claim 10, further comprising: a restartdetermination unit that determines a restart of an operation of theregion acquisition unit, wherein when the restart determination unitdetermines a restart of an operation, the region